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Key findings 



This study examined the survey responses of teachers 
from 75 Idaho schools working on school improvement. 
The schools with higher teacher reports of the presence 
f the goals, processes, and supports essential for 
student success did not have higher rates of reading 
proficiency, math proficiency, or attendance. 
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Summary 


Policymakers and practitioners frequently use teacher surveys to inform decisions on 
school improvement efforts in low-achieving schools. But there is little empirical evidence 
on how the results of these surveys relate to student outcomes. This study provides infon 
mation on how perception data from a teacher survey in Idaho is correlated with three 
student outcomes: reading proficiency, math proficiency, and attendance. 

The Idaho State Department of Education uses the Educational Effectiveness Survey 
(EES), an annual teacher survey developed and administered by the Center on Educa- 
tional Effectiveness, to gather information on school qualities believed to be the goals, 
processes, and supports essential for school success. Used widely in the Northwest Region, 
the survey is similar to teacher perception surveys used nationally. This study covers the 
75 low-achieving Idaho schools that used the state’s school improvement services and took 
the EES in 2012. 

The analyses of the survey data revealed that teachers’ perceptions of the presence of 
essential goals, processes, and supports were generally not related to reading proficiency, 
math proficiency, or attendance. A few significant relationships were found in subsamples 
of schools. For example, the essential support defined in the survey as “effective school 
leadership” was significantly related to reading proficiency in 2011 in the 33 schools with 
data for 2010, 2011, and 2012. A significant positive relationship was also found between 
school attendance in elementary schools in 2012 and teacher ratings of five of nine other 
essential goals, supports, and processes. 

The weak relationship between teacher perceptions on the EES and student outcomes does 
not support the use of the EES as an indicator of academic progress in Idaho’s low-achiev- 
ing schools, particularly not as the sole indicator. Other uses of perception data from the 
EES — such as measuring teacher satisfaction with school environments — may be useful 
for practitioners but were not examined in this study. 

The findings suggest that Idaho educators and others using teacher perception surveys 
should proceed cautiously in making decisions based on perception surveys. For example, 
researchers and policymakers should consider how well survey measures relate to desired 
student outcomes and provide useful information on the effectiveness of school improve' 
ment efforts. They might also consider using data from other sources to assess the school 
goals, processes, and supports they seek to investigate through teacher surveys. 
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Why this study? 


Many education practitioners and policymakers use teacher surveys to track progress and 
evaluate the success of school improvement efforts. Such surveys range from district-ad- 
ministered surveys in large districts such as Chicago (Bryk, Sebring, Allensworth, Lup- 
pescu, & Easton, 2010) and New York (McCormick & Nathanson, 2013) to commercial 
surveys intended to inform state decisions on school assistance (Center for Educational 
Effectiveness, 2008) to surveys used primarily for school accreditation (Weaver & Barile, 
2011). But little research has explored how well the results of teacher surveys correlate 
with the student outcomes that school improvement efforts seek to affect. (Appendix A 
provides detailed information on research related to this study.) 

This study examines whether teacher ratings of school goals, processes, and supports on 
the Educational Effectiveness Survey (EES; box 1) correlate with three key outcomes 
in Idaho’s low-achieving schools: reading proficiency, math proficiency, and attendance. 
(Box 2 looks at how the Idaho State Department of Education uses the EES; box 3 and 
appendix B detail the data sources and methods used in the study.) 


Box 1. About the Educational Effectiveness Survey 

The Center for Educational Effectiveness developed the Educational Effectiveness Survey 
based on a research synthesis conducted by school improvement specialists at the Wash- 
ington Office of Superintendent of Public Instruction in 2003 and revised in 2007 (Shannon 
& Bylsma, 2003, 2007). That synthesis identified nine dimensions as the essential goals, 
processes, and supports needed for school success: 

• Clear and shared focus. All teachers and staff know where they are going and why, based 
on a focus on a shared vision and an understanding of their role in realizing it. The focus 
and vision are derived from common beliefs and values, creating a consistent direction for 
all involved. (Sample item: The school has a clear sense of purpose.) 

• High standards and expectations for all students. Teachers and staff believe that all stu- 
dents can learn and meet high standards. While some students must overcome sub- 
stantial barriers, the barriers are not seen as insurmountable. Students are offered an 
ambitious and rigorous course of study. (Sample item: All students are expected to achieve 
high standards.) 

• Effective school leadership. Effective instructional and administrative leadership is 
required to implement change. Effective leaders seek needed help and nurture an instruc- 
tion program and school culture conducive to learning and professional growth. Effective 
leaders have different styles and roles; teachers and other staff members, including those 
in the district office, often have a leadership role. (Sample item: People in leadership roles 
act with integrity.) 

• High levels of collaboration and communication. There is strong teamwork among teach- 
ers across all grades and with other staff. Everyone is involved and connected — including 
parents and other members of the community — in identifying problems and working on 
solutions. (Sample item: Teachers discuss teaching issues on a regular basis.) 

• Curriculum, instruction, and assessment aligned with standards. The planned and actual cur- 
ricula align with the essential academic learning requirements. Research-based teaching 
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Box 1. About the Educational Effectiveness Survey (continued) 

strategies and materials are used. Staff members understand the role of classroom and 
state assessments, what the assessments measure, and how student work is evaluated. 
(Sample item: The school’s curriculum aligns with state standards.) 

• Frequent monitoring of learning and teaching. A steady cycle of different assessments iden- 
tify students who need help. These students receive more support and instruction time, 
either during the school day or at other times. Teaching is adjusted based on frequent 
monitoring of student progress and needs. Assessment results are used to focus and 
improve instruction programs. (Sample item: Students receive regular feedback about 
what they need to do to improve.) 

• Focused professional development. A strong emphasis is placed on training staff in areas 
with the most need. Feedback from learning and teaching focuses on extensive and 
ongoing professional development. The support aligns with the school or district vision 
and objectives. (Sample item: Assessment results are used to determine professional 
learning activities.) 

• Supportive learning environment. The school has a safe, civil, healthy, and intellectually 
stimulating learning environment. Students feel respected and connected with the staff 
and are engaged in learning. Instruction is personalized, and small learning environments 
increase student contact with teachers. (Sample item: Students feel safe on school prop- 
erty during school hours.) 

• Fligh levels of family and community involvement. There is a sense that all involved — 
families, businesses, social service agencies, and community colleges and universities — 
have a responsibility to educate students, not just teachers and school staff. (Sample 
item: The staff believes that students learn more through effective family support.) 
Shannon and Bylsma’s literature review (2007) suggests that all nine dimensions are 

equally important and must be addressed simultaneously to promote school improvement. 

(The full survey can be found at http://extranet.educationnorthwest.org/sites/default/files/ 

nine-characteristics-survey.pdf.) 


Box 2. About the Idaho State Department of Education’s use of the Educational 
Effectiveness Survey 

In 2008 the Idaho State Department of Education based a major overhaul of its support system 
on the goals, processes, and supports essential for student success discussed in box 1. The 
No Child Left Behind Act of 2001 requires states to identify schools in need of improvement 
based on a failure to make adequate yearly progress toward targets set by the state for two or 
more consecutive years, to create a statewide support system for schools with longer histories 
of failing to meet these targets, and to identify distinguished educators who can help deliver 
these supports (No Child Left Behind Act of 2001, 2002). Several studies have reported on 
the limited capacity of states to provide this technical assistance (Edwards, 2006; Le Floch, 
Boyle, & Therriault, 2008; Minnici & Hill, 2007), and Idaho is one of many states continuing to 
refine its system. 

When a school enters improvement status, the Idaho State Department of Education 
offers up to three years of capacity-building services (on-site and remote technical assistance 
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Box 2. About the use of the Educational Effectiveness Survey by the Idaho State 
Department of Education (continued) 

and professional development from retired educators recognized as leaders and experts in 
school improvement). Providers of capacity-building services work with principals and school 
leadership teams on school improvement plans to increase the staff-reported presence of tar- 
geted school goals, processes, and supports (S. Underwood, director of the Idaho statewide 
system of support, personal communication, April 12, 2012). 

To evaluate these capacity-building efforts and to pinpoint which services to provide in 
each school, the state hired the Center for Educational Effectiveness to administer its multi- 
item Educational Effectiveness Survey (EES) annually, beginning in 2008, to all schools that 
receive such services (Lane, 2010; Underwood, 2013). The school-based reports show the 
mean score for each item as rated by teachers. For example, a survey that shows low teacher 
ratings for “collaboration and communication among staff members" might suggest that the 
school improvement plan call for more team planning time and professional development in 
effective communication strategies. 

These components — capacity-building services and the administration of the EES — 
remain the basis of Idaho’s school support system. A 2010 external case study confirmed that 
research on essential school goals, processes, and supports did indeed inform the capacity- 
building services and that the service providers reported using this framework in their work 
with schools and districts (Lane, 2010). Several Idaho school and district officials have tes- 
tified to the usefulness of capacity-building services (Scott, McMurrer, McIntosh, & Dibner, 
2012 ). 

Although the goals, processes, and supports identified in Shannon and Bylsma (2003, 
2007) and measured through the EES are frequently found in higher achieving schools, the 
evidence of the relationship between use of EES measures and long-term student outcomes 
such as test scores and attendance remains limited. 


Box 3. Data sources and methods 

This study used two main data sources: the Educational Effectiveness Survey (EES) and pub- 
licly available data from the Idaho State Department of Education. The survey data included 
teacher responses on the 2012 EES for 75 schools designated by the Center for Education- 
al Effectiveness as receiving capacity-building services. The study also examined data for a 
subset of 33 schools that had received services and participated in the EES for three years 
(2010, 2011, and 2012). Of the 107 items on the EES (version 9), 86 measured school staff 
members’ perceptions of the presence of nine dimensions related to school goals, process- 
es, and supports. All teachers in the 75 schools were invited to participate: 1,745 teach- 
ers responded in 2012 (91 percent of teachers). For the subset of 33 schools 820 teachers 
responded in 2010 (87 percent), 830 responded in 2011 (86 percent), and 863 responded in 
2012 (89 percent; see table B2 in appendix B). No school with a below-average return rate had 
fewer than four teachers who responded. 

The publicly available school-level information from the Idaho State Department of Edu- 
cation included data on reading proficiency, math proficiency, attendance, and percentage of 
students eligible for free or reduced-price lunch for 2010, 2011, and 2012. Reading and math 
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Box 3. Data sources and methods (continued) 


proficiency and attendance were used as student outcome measures; the percentage of stu- 
dents eligible for free or reduced-price lunch was used as a proxy for poverty at each school. 1 
More than 90 percent of schools had data for all variables (see table B3). Attendance data 
were available for only 2010 and 2012; the 2011 data were lost during a database conver- 
sion. In general, average proficiency rates rose slightly from 2010 to 2012, as did teachers’ 
average self-reports on most of the nine dimensions (see table C5 in appendix C). Attendance 
remained stable. Although other research on the EES does not use attendance as a variable 
(Bylsma, 2008; Thacker & Becker, 2012), this study did, because other research suggests 
that attendance frequently influences performance (Conard, 2006; Duran-Narucki, 2008). 

The data were analyzed in three steps: descriptive analyses, including examination of 
scatterplots and histograms; correlations between EES data and reading and math proficien- 
cy and student attendance; and regression analyses using reading and math proficiency and 
student attendance as outcome variables, the percentage of students eligible for free or 
reduced-price lunch as a covariate, and each EES dimension as the independent variable. Both 
correlations and regressions used the Benjamini-Hochberg correction to adjust for multiple 
comparisons. Separate analyses were also conducted for elementary schools (n = 35) and 
secondary schools (n = 39). One school did not fall into either category. Appendix B provides 
more detail on data sources and methods. 

1. The National School Lunch Program provides free lunch for students from households at or below 130 per- 
cent of the poverty line and reduced-price lunch for students from households at 131-185 percent of the pov- 
erty line. Poverty lines are established and updated by the U.S. Department of Health and Human Services. 


For the 75 schools 
receiving capacity- 
building services 
in 2012, 28 of the 
30 correlations 
were positive; 
however, none of 
the correlations 
was statistically 
significant 


Study findings 


This section discusses the study’s three key findings on the relationship between student 
outcomes (reading and math proficiency and attendance) and teacher ratings on the EES 
of school goals, processes, and supports. 

No relationship between student outcomes and teacher ratings of school goals, processes, and 
supports for the full sample of schools in 2012 

For the 75 schools receiving capacity-building services in 2012, 28 of the 30 correlations 
were positive (table 1). (Schools with higher mean scores on the EES also had higher 
mean proficiency and attendance rates.) However, none of the correlations was statistically 
significant. 1 

The relationship between teacher survey data from the EES and the publicly available 
outcome data for the 75 schools in 2012 was further examined through a series of 30 linear 
regression analyses at the school level. In each linear regression the dependent variable 
was one of the three schooHevel outcomes (reading proficiency rates, math proficiency 
rates, or attendance rates), and the variable of interest was 1 of the 10 survey dimensions 
(nine dimensions and the mean of all dimensions). The percentage of students eligible 
for free or reduced-price lunch was a covariate that served as a proxy controlling for the 
level of need of the student body, because several studies have confirmed the link between 
schooHevel percentages of students eligible for free or reduced-price lunch and student 
outcomes (Kahlenberg, 2001). No EES dimension was found to be significantly related to 
an outcome of interest (table 2). 
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Table 1. No statistically significant correlations between school-level Educational 
Effectiveness Survey data and student outcomes for all schools, 2012 



Pearson product-moment correlation coefficients 

Dimension 

Percentage 
of students 
proficient in 
reading (n = 73) 

Percentage 
of students 
proficient in 
math (n = 73) 

Student 
attendance 
rate (n = 70) 

Frequent monitoring of learning and teaching 

.11 

.24 

.20 

Supportive learning environment 

.08 

.13 

.21 

Effective school leadership 

.07 

.07 

.16 

High levels of collaboration and communication 

.05 

.10 

.23 

High levels of family and community involvement 

.05 

.05 

.19 

Focused professional development 

.04 

.14 

.22 

Curriculum, instruction, and assessment aligned 
with standards 

.02 

.20 

.23 

Clear and shared focus 

.00 

.10 

.17 

High standards and expectations for all students 

-.17 

-.03 

.16 

Total items 

.04 

.13 

.22 


Note: Of the 75 schools in the sample, 2 were missing data on reading proficiency, 2 were missing data on 
math proficiency, and 5 were missing data on attendance. Each cell represents the estimated correlations 
between the EES dimension for that cell’s row (row header) and the student outcome for that cell’s column 
(column header). Rows are ordered by the largest correlation for reading. 

Source: Authors’ analysis of data from the Educational Effectiveness Survey (2012). 


None of the 30 
correlations for 
secondary schools 
was statistically 
significant. 


No statistically significant relationships in elementary or secondary schools between teacher ratings 
and reading or math proficiency, but some between teacher ratings and attendance for elementary 
schools 

Correlations were stronger for elementary schools than for secondary schools (table 3). 
However, as with the complete sample, no statistically significant relationships were found 
between teacher ratings and reading or math proficiency rates. Of the 10 relationships 
tested (nine dimensions and the mean of all dimensions), only 6 showed statistically sig- 
nificant correlations between teacher ratings and student attendance rates for elementary 
schools: 

• Effective leadership (r 2 = .47). 

• Frequent monitoring of learning and teaching (r 2 = .46). 

• Focused professional development (r 2 = .46). 

• High levels of collaboration and communication (r 2 = .41). 

• Supportive learning environment (r 2 = .40). 

• Mean for all dimensions (r 2 = .43). 

None of the 30 correlations for secondary schools was statistically significant. And 26 of 
the 30 were negative, meaning that schools with lower average EES ratings had higher pro- 
ficiency and attendance rates, though these correlations could have occurred by chance. 

One statistically significant relationship between teacher ratings and student outcomes for the 
subsample of 33 schools that conducted the survey in 2010, 2011, and 2012 

The only statistically significant relationship found for the subsample of 33 schools with 
data for three years was between effective school leadership and reading proficiency in 
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Table 2. Results of linear regressions showed no significant relationships 


Linear regression correlation 
coefficients (standard errors) 

Dimension 

Reading 
(n = 73) a 

Math 
(n = 73) a 

Attendance 
(n = 70) b 

Frequent monitoring of learning and teaching 

0.020 

(0.02) 

0.066 

(0.03) 

0.028 

(0.02) 

Supportive learning environment 

0.013 

(0.03) 

0.040 

(0.04) 

0.037 

(0.02) 

High levels of collaboration and communication 

0.011 

(0.02) 

0.029 

(0.04) 

0.037 

(0.02) 

Effective school leadership 

0.011 

(0.02) 

0.017 

(0.03) 

0.023 

(0.02) 

Focused professional development 

0.008 

(0.02) 

0.038 

(0.03) 

0.032 

(0.02) 

High levels of family and community involvement 

0.004 

(0.02) 

0.007 

(0.03) 

0.028 

(0.02) 

Curriculum, instruction, and assessment aligned with standards 

0.002 

(0.02) 

0.058 

(0.04) 

0.039 

(0.02) 

Clear and shared focus 

0.002 

(0.03) 

0.039 

(0.05) 

0.034 

(0.03) 

High standards and expectations for all students 

-0.028 

(0.02) 

-0.005 

(0.03) 

0.024 

(0.02) 

Mean of all dimensions 

0.008 

(0.03) 

0.041 

(0.04) 

0.040 

(0.02) 


a. 70 degrees of freedom. 

b. 67 degrees of freedom. 

Note: Of the 75 schools in the sample, 2 were missing data on reading proficiency, 2 were missing data 
on math proficiency, and 5 were missing data on attendance. Each cell presents the coefficient (and the 
coefficient’s standard error) from a separate linear regression in which the outcome was the school’s reading 
proficiency rate, math proficiency rate, or attendance rate (column headers); the predictor was one of the nine 
dimensions (row headers); and the covariate was the percentage of students eligible for free or reduced-price 
lunch (not shown in table). Rows are ordered by the largest correlation for reading. 

Source: Authors' analysis of data from the Educational Effectiveness Survey (2010, 2011, and 2012) and data 
from Idaho State Department of Education (n.d. a, b, c, d). 


2011 (r 2 = .49; table 4)- Correlations with reading proficiency rates were all positive in 2010 
and 2011 but negative for eight dimensions in 2012, though none was statistically signifi- 
cant. As with reading, math proficiency rates had more negative correlations in 2012 than 
in 2010 and 2011, though none was statistically significant. 

Implications of the findings 


This study found no relationship between Idaho teacher ratings of school processes, goals, 
and supports as measured by the EES and reading proficiency, math proficiency, or atten- 
dance in the full sample of 75 schools. Therefore, there is no evidence to support the use 
of the average teacher ratings on the nine EES dimensions to track progress in student 
achievement in Idaho schools that receive capacity-building services. The lack of relation- 
ships also casts doubt on the utility of EES data for assessing the overall impact of the 
Idaho statewide system of support. Idaho policymakers are strongly advised to consider 
an alternative method of measuring school processes, goals, and supports — one that is 
more strongly associated with long-term student outcomes. This suggestion is consistent 
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Table 3. Few statistically significant correlations between dimensions of the 
Educational Effectiveness Survey and student outcomes, by grade-level groupings, 
2012 




Pearson product-moment correlations 



Percentage of students 
proficient in reading 

Percentage of students 
proficient in math 

Student attendance 
rate 

Dimension 

Elementary 
schools 
(n = 35) 

Secondary 
schools 
(n = 37) 

Elementary 
schools 
(n = 35) 

Secondary 
schools 
(n = 37) 

Elementary 
schools 
(n = 34) 

Secondary 
schools 
(n = 35) 

Frequent monitoring of 
learning and teaching 

.31 

-.01 

.29 

.02 

46** 

-.10 

High levels of collaboration 
and communication 

.27 

-.15 

.24 

-.11 

41** 

.04 

Supportive learning 
environment 

.26 

-.10 

.20 

-.02 

40** 

-.01 

High levels of family and 
community involvement 

.24 

-.20 

.15 

-.19 

.32 

.02 

Clear and shared focus 

.22 

-.21 

.17 

-.22 

.27 

-.04 

Effective school leadership 

.20 

-.05 

.07 

-.04 

.47** 

-.10 

Curriculum, instruction, and 
assessment aligned with 
standards 

.19 

-.09 

.13 

-.02 

.24 

.11 

Focused professional 
development 

.19 

-.03 

.15 

-.08 

46** 

-.06 

High standards and 
expectations for all 
students 

-.05 

-.29 

-.05 

-.28 

.29 

-.12 

Total items 

.23 

-.12 

.17 

-.10 

.43** 

-.03 

** Statistically significant at the < .05 level, using the Benjamini-Hochberg correction. 

Note: Each cell represents the estimated correlations between the Educational Effectiveness Survey dimen- 
sion for that cell’s row (row header) and the student outcome by secondary or elementary school for that cell’s 
column (column header). Rows are ordered by the largest correlation in elementary schools in reading. 

Source: Authors' analysis of data from the Educational Effectiveness Survey (2012). 


This study suggests 
that the predictive 
validity of such 
measures as the 
EES needs to 
be established 
along with 
content validity 


with recommendations for practice from other measurement experts (Schmeiser & Welch, 
2006). Without evidence of efficacy for tracking progress and evaluating school improve- 
ment efforts, teacher surveys have uncertain value in decisionmaking for school improve- 
ment efforts. 2 

This study also has implications for researchers and policymakers developing and using 
perception surveys on school factors that influence student outcomes. Several recent 
studies, for example, point to the use of perception surveys in high-stakes decisions, includ- 
ing on principal evaluation (Clifford, Menon, Gangi, Condon, & Hornung, 2012; Hal- 
linger, Wang, & Chen, 2013), school accreditation (Weaver & Barile, 2011), and school 
accountability (McCormick &Nathanson, 2013). 

Whatever survey researchers and policymakers use, it is recommended that they exercise 
caution when using the results to make decisions. Survey developers typically establish the 
content validity of perception surveys with literature reviews and expert consultations, but 
few surveys show evidence of predictive validity by testing how well their dimensions relate 
to other outcomes (Clifford et al., 2012). This study suggests that the predictive validity of 
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Table 4. Few statistically significant correlations between dimensions of the 
Educational Effectiveness Survey and student outcomes in the 33 schools with 
data for 2010, 2011, and 2012 


Pearson product-moment correlations 
Percentages of Percentages of Attendance rates 

students proficient students proficient (n = 31 in 2010, 
in reading (n = 32) in math (n = 32) n = 29 in 2012) 


Dimension 

2010 

2011 

2012 

2010 

2011 

2012 

2010 

2011 

2012 

High levels of collaboration and 
communication 

.41 

.34 

-.02 

.15 

.08 

-.11 

.33 



.16 

Frequent monitoring of learning 
and teaching 

.41 

.40 

.04 

.29 

.15 

-.04 

.22 



.20 

Supportive learning environment 

.40 

.40 

-.06 

.28 

.20 

-.10 

.26 

— 

.12 

Focused professional development 

.37 

.31 

.02 

.28 

.41 

-.07 

.31 

— 

.30 

Clear and shared focus 

.34 

.20 

-.15 

.22 

.06 

-.12 

-.04 

— 

.11 

Curriculum, instruction, and 
assessment aligned with 
standards 

.33 

.38 

-.15 

.22 

.09 

-.09 

-.18 


.10 

High standards and expectations 
for all students 

.29 

.18 

-.26 

.26 

.04 

-.19 

.28 



.08 

High levels of family and 
community involvement 

.27 

.38 

-.10 

.20 

.24 

-.08 

.01 



.25 

Effective school leadership 

.21 


-.08 

.18 

.14 

-.26 

.23 

— 

.21 

Total items 

.42 

.41 

-.08 

.28 

.18 

-.13 

.21 

— 

.18 


— is not available because data were lost in a data storage system conversion. 

** Statistically significant at the < .05 level, using the Benjamini-Hochberg correction. 

Note: Of the 33 schools in the sample, 1 was missing data on reading proficiency, 1 was missing data on 
math proficiency, 4 were missing data on attendance in 2010, and 2 were missing data on attendance in 
2012. Each cell represents the estimated correlations between the Educational Effectiveness Survey dimen- 
sion for that cell’s row (row header) and the student outcome by year for that cell’s column (column header). 
Rows are ordered by 2010 reading proficiency rates. 

Source: Authors' analysis of data from the Educational Effectiveness Survey (2010, 2011, and 2012) and data 
from Idaho State Department of Education (n.d. a, b, c, d). 


such measures as the EES needs to be established along with content validity. Researchers 
and policymakers relying on teacher perception surveys for decisionmaking should consid- 
er predictive validity to ensure that surveys provide useful information on school effective- 
ness that is related to student outcomes. 

Given these limitations of perception surveys, researchers and policymakers in Idaho and 
across the country might consider multiple other measures of the school goals, processes, 
and supports they seek to investigate through teacher surveys. Some existing tools might 
be adapted, or new tools might be developed. For example, the No Child Left Behind Act 
of 2001 — and its flexibility guidance — requires school improvement plans from schools 
identified as low achieving (U.S. Department of Education, 2012). A tool measuring the 
presence and implementation of the school goals, processes, and supports as described in 
these plans could prove useful, especially in the many states, such as Idaho, that have 
adopted online school improvement planning and tracking systems. Additional tools, 
such as observation protocols used by states as part of monitoring low-achieving schools, 
might also provide useful information on school goals, processes, and supports that could 
be linked to student outcomes. The number of states that report these monitoring data has 
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increased due to new federal requirements for school improvement grants (McMurrer & 
McIntosh, 2012). Ultimately, an analysis that combines multiple measures of school goals, 
processes, and supports and then investigates their relationship with formative and sum- 
mative assessments could help educators and policymakers strengthen school improvement 
efforts. 


Limitations of the study 


This study has several limitations. First, this is a correlational study; the results do not 
support causal claims about the impact of state services or about whether changes in school 
goals, processes, and supports result in changes in student outcomes. Examining causality 
is beyond the scope of this study. This study is instead a first step toward understanding 
the utility of a teacher perception survey as a predictor of student achievement. 

Second, the study did not examine other uses of the EES, such as measuring teacher 
satisfaction. 

Third, the correlations and regression models had restricted ranges for the variables. This 
restriction was due in part to aggregation to the school level as a result of the Idaho State 
Department of Education’s interest in schoohlevel capacity building. Some outliers were 
included in the analyses, either because no pattern could be identified for them or because 
it was assumed that Idaho leaders wanted to build capacity in these types of schools (alter' 
native schools) specifically. For the regression analyses, the models did not consider that 
student outcomes could affect teacher ratings. The models also do not account for all mea- 
surement error in the EES. Therefore, these results must be interpreted with caution. 

Fourth, the study examined only one measure of school goals, processes, and supports: 
teacher ratings on the EES. Examining other indicators, such as differences in the quality 
of school improvement plans or direct observation, might reveal differences across schools 
that did not emerge from the EES ratings. Comparing multiple measures of school goals, 
processes, and supports would also help determine the validity of the EES. 

Fifth, the study did not gather detailed information on how educators in Idaho use the 
EES or the extent to which its use depends on positive correlations between the EES and 
student outcomes. For example, educators may find the EES useful in planning school 
improvement due to factors other than the correlations this study examines. 

Sixth, the study cannot directly explain why correlations between the EES dimensions 
and attendance differ from elementary to secondary schools. Results may differ for several 
reasons. Parents might have more direct control over student attendance in elementary 
school than in secondary school. Elementary school teachers might be more (or less) accu- 
rate in their EES responses. The EES may be more (or less) accurate in measuring the nine 
dimensions in elementary schools. Or some other variable that the study was unable to 
account for might affect the relationship. 

Finally, the study cannot comment on the theoretical validity of the nine dimensions mea- 
sured in the EES as levers for improving student outcomes. Other dimensions representing 
school goals, processes, and supports might also have merit. This study chose to focus on 
the EES because it is the measure used most frequently in Idaho. 


This study is a 
first step toward 
understanding the 
utility of a teacher 
perception survey 
as a predictor 
of student 
achievement 
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Appendix A. Connections to previous research 


Educators have long known that the quality of schools and teachers makes a difference for 
students. Beginning in the 1990s, researchers began developing models to account for the 
value that teachers add to student achievement (Sanders & Horn, 1994). 

The impact of schools on student achievement has also been of interest to educators. In 
a meta-analysis of 10 rigorous studies that attempted to isolate the impact of schools on 
student achievement, Marzano (2000) found that schools accounted for about 20 percent 
of the variance in the models. Given that the state test passage rate was expected to be 
50 percent, Marzano determined that the average rate for an effective school (that is, a 
school that performs better than would be predicted based on its demographic dimensions) 
is 72 percent, compared with 28 percent for an ineffective school. A recent international 
study confirmed the importance of schooHevel impacts on student achievement in a wide 
range of countries (Willms, 2010). 

Several researchers have attempted to codify the factors that influence a school’s impact 
on student outcomes (for example, Cotton, 1999; Edmonds, 1979; Lezotte, 1991). The 
research synthesis conducted by school improvement specialists at the Washington Office 
of Superintendent of Public Instruction is a continuation of this line of inquiry (Shannon 
& Bylsma 2003, 2007). In the 2003 synthesis the research team reviewed 25 articles to 
identify common school dimensions associated with effective schools. 

The studies reviewed by Shannon and Bylsma varied considerably in the methods used 
to identify effective schools, and none was experimental. For example, Barth et al. (1999) 
asked states to submit a list of their top-scoring, high-poverty schools, while the authors of 
a study from the Education Commission of the States (1999) reflected on their own experi- 
ences and lessons learned working with governors and state and local policymakers to raise 
awareness of comprehensive school reform. 

In 2006 the Washington Office of Superintendent of Public Instruction convened a panel 
to validate the Shannon and Bylsma synthesis and to help the state further define the 
goals, processes, and supports essential for effective schools. As a result of the panel’s input, 
the literature review added 120 references (Shannon & Bylsma, 2007). The new compi- 
lation included earlier comprehensive reviews of the literature (Cotton, 1999; Edmonds, 
1979; Lezotte, 1991) and added other more sophisticated studies, such as Bryk and Schnei- 
der (2002), which used multilevel regression models to identify schools that performed 
better than a comparison group. 

But what measures of the school dimensions accurately predict higher student achievement? 
To answer this question, scholars have created and tested perception survey instruments 
that measure some of the subjective school factors that might impact student achievement. 
Because school staff control these subjective factors, this research is particularly relevant 
to school improvement work. 

Developers typically design perception surveys with the goal of ensuring that the surveys 
are valid (that they measure the ideas they intend to measure) and reliable (that they 
measure these ideas the same way over time). Next, they test perception surveys for valid- 
ity and reliability using a variety of methods. These methods sometimes include tests of 
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predictive validity, which evaluate how the survey results correspond to other measures of 
the ideas or factors the survey measures (Fowler, 2009). 

A recent review of teacher perception surveys used for principal evaluation found 13 surveys 
that met the authors’ criteria for validity and reliability (Clifford et al., 2012). However, just 
three of the instruments examined the predictive validity of their perception surveys (the 
survey measure’s correlation with desirable outcomes, such as improved student test scores 
and attendance). 

Only one study, a dissertation using Washington state data, has examined and found signif- 
icant relationships between average teacher EES ratings and student achievement (Bylsma, 
2008). Appendix B examines the Washington study and the validity and reliability of the 
EES. The current study adds to information on teacher perception surveys in general and 
to knowledge about the predictive validity of the EES in Idaho in particular. 
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Appendix B. Data sources and methods 


This appendix provides further details on the Educational Effectiveness Survey, publicly 
available data from the Idaho State Department of Education, participating schools, data 
collection, and study methods. 

The Educational Effectiveness Survey 

Description and scoring. The Center for Educational Effectiveness (CEE) created the Edu- 
cational Effectiveness Survey (EES) based on the work of Shannon and Bylsma (2003). 
Of the 107 items on the EES (version 9), 86 measure school staff members’ reports on the 
presence of school goals, processes, and supports as represented by the nine EES dimem 
sions (table Bl). This study used teachers’ responses, rather than all staff responses, for two 
reasons. First, because teachers have the strongest response rate, as compared with non- 
certified staff and administrators, and second, because teachers have the most contact with 
the students whose outcomes the study examined. CEE has administered the survey in 
Idaho schools by mail, annually from 2008 through 2012, and plans to continue to do so. 
Idaho schools participating in the survey include those receiving capacity-building services 
from the Idaho State Department of Education. 

Respondents rated EES items on a scale of 1 to 5, where 1 was “almost never true,” 2 was 
“seldom true,” 3 was “sometimes true,” 4 was “often true,” and 5 was “almost always true.” 
The original numeric coding of the scale was reversed for this study, so that positive cor- 
relations indicate that higher means on the EES are related to higher student proficiency 
and attendance rates. Respondents could also select “no opinion,” but only one respondent 
did and only for one item; this single response was treated as missing. Each item included 
in this study relates to one of the nine EES dimensions. The full survey can be found at 
http://extranet.educationnorthwest.org/sites/default/files/nine'characteristicS'Survey.pdf. 


Table Bl. Sample items for each dimension of the Educational Effectiveness Survey 

Dimension 

Number 
of items 

Sample item 

Clear and shared focus 

7 

The school has a clear sense of purpose. 

High standards and expectations for all 
students 

7 

All students are expected to achieve high 
standards. 

Effective school leadership 

10 

People in leadership roles act with integrity. 

High levels of collaboration and communication 

12 

Teachers discuss teaching issues on a regular 
basis. 

Curriculum, instruction, and assessment 
aligned with standards 

10 

The school’s curriculum is aligned with state 
standards. 

Frequent monitoring of learning and teaching 

8 

Students receive regular feedback about what 
they need to do to improve. 

Focused professional development 

10 

Assessment results are used to determine 
professional learning activities. 

Supportive learning environment 

16 

Students feel safe on school property during 
school hours. 

High levels of family and community 
involvement 

6 

The staff believes students learn more through 
effective family support. 

Source: Center for Educational Effectiveness, 2008. 
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Validity and reliability. CEE administered version 9 of the EES in Idaho as part of the state’s 
school improvement initiative, and Regional Educational Laboratory Northwest used the 
survey data for this study. (A version 10 is now available.) CEE has done several things to 
help ensure the validity and reliability of the EES. However, additional work (by CEE or 
others) might be needed. This section describes the work that has been done, the work 
that remains to be done, and how the current study contributes to efforts to examine the 
validity of the EES. 

As mentioned, CEE originally developed the EES based on Shannon and Bylsma (2003), as 
well as on organizational effectiveness research and expert opinion. During 2003-09 CEE 
strengthened the content validity of the EES by revising the survey in response to recent 
developments in school effectiveness research. In addition, it factored client feedback into 
each revision. Through relationships with leading educators, professional associations, and 
state departments of education, the EES underwent five revisions between 2003 and 2009, 
each bringing the latest in research-based and professionally grounded understanding 
to the instrument. However, according to CEE’s chief executive officer, the nine survey 
dimensions have remained essentially the same since the third version in 2004, making it 
possible to construct previous-year dimensions that would parallel the 2012 administration 
(version 9) used in this study (G. Lobdell, chief executive officer, personal communication, 
May 14, 2012). 

With each version, CEE also did cognitive pretesting. For version 9, CEE conducted cog- 
nitive pretesting in a focus group with staff members from three Washington state school 
districts and in a peer review with education researchers who were former members of the 
CEE board of directors when CEE was a nonprofit organization. Based on this input, CEE 
refined version 9 and piloted the survey. The organization typically pilots each new version 
in at least 12 buildings, involving at least 600 staff members (G. Lobdell, chief executive 
officer, personal communication, January 31, 2013). 

In addition, CEE assesses the internal consistency of each of the nine dimensions for 
each EES administration using Cronbach’s alpha. This statistic ranges from 0 to 1 and 
represents the intercorrelation of items intended to measure the same dimension. Stronger 
correlations mean a greater likelihood that the items measure the same dimension. The 
alphas for past administrations of the EES ranged from .75 to .94 for version 9 (Center 
for Educational Effectiveness, n.d.). Alphas for all EES dimensions were greater than .70, 
which many researchers consider the minimum to justify using an instrument to measure 
particular dimensions (Hair, Black, Babin, Anderson, & Tatham, 2006). 

However, some questions remain about the validity of the EES. For example, all items in 
version 9 are worded positively. When comparing positively worded to negatively worded 
items in 2001 and 2002, CEE found no differences in the Cronbach’s alphas for the dimen- 
sions, but negatively worded items created another step in data analysis and reporting, 
as well as some confusion for clients reading the reports. Therefore, CEE decided to use 
only positively worded items (G. Lobdell, chief executive officer, personal communication, 
January 31, 2013). This decision could cause participants to focus more on positive respons- 
es, limiting the range of the scale and making its validity questionable (Malhotra, 2006). 

Other questions about validity could be addressed by comparing the EES to other measures 
of school goals, processes, and supports — something CEE has not yet done. An evaluation 
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report in Washington state that used both the EES and classroom observations showed 
that the results varied widely across the eight participating districts. The study did not 
attempt to compare the two measures, but the authors noted that observations generally 
had higher ratings for the second set of schools that participated in the study, while the 
reverse was true of the EES ratings (Thacker & Becker, 2012). 

Finally, the predictive validity of the EES needs further exploration. Predictive validity is 
how a measure such as the EES correlates with other related measures (such as other per- 
ception surveys) or with the ultimate outcomes of interest (such as student outcomes). The 
current study contributes to the need to explore the predictive validity of the EES. 

The literature review conducted for this study identified just one study that examined the 
predictive validity of the EES (see appendix A). That one study found statistically signif- 
icant correlations between survey dimensions and student achievement (Bylsma, 2008). 
The author conducted the study in Washington state and used Pearson’s correlations 
between the school means for each of the nine EES dimensions and schooUevel percent- 
ages of proficient students on state math and reading tests for three grade levels: elementa- 
ry, middle, and high school. 

For elementary schools seven dimensions showed statistically significant correlations 
(.252— .462), meaning that variations in EES scores accounted for 6-21 percent of the vari- 
ance in percentages of proficient students (calculated by squaring the Pearson’s correla- 
tion, as Marzano [2000] did). These percentages represent the relationship between the 
variance in the two variables and should not be misinterpreted as indicating that either 
variable caused the other variable to vary in any way. In middle schools eight dimensions 
showed statistically significant correlations (.213-559), accounting for 5-31 percent of the 
variance. In both elementary and middle schools the largest correlations were between 
the EES dimension for family and community engagement and the composite math and 
reading proficiency rates. Determining how actual changes in the EES related to changes 
in student test scores over time was beyond the scope of Bylsma (2008). 

In high schools no statistically significant correlations were found (Bylsma, 2008). The 
author speculated that this may be due to the fact that EES scores for high school were 
lower and contained less variation than those for other grade levels and because the sample 
size was smaller for high schools (n = 70) compared with elementary schools (n = 207) and 
middle schools (n = 102; Bylsma, 2008). Another factor may be that the state high school 
test is given in Washington state only at the end of grade 10, when some students have 
already dropped out. This may result in a test that is not as representative as the one given 
to elementary and middle school students, whose attendance is still compulsory. 

Idaho State Department of Education publicly available data 

The second data source for this study was publicly available information from the Idaho 
State Department of Education, including school-level proficiency rates on state test scores 
(Idaho State Department of Education, n.d. b), attendance records (Idaho State Depart- 
ment of Education, n.d. c.), and percentage of students eligible for free or reduced-price 
lunch (Idaho State Department of Education, n.d. d). These data were used to represent 
student achievement and poverty levels in the analyses. 
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Idaho’s state reading and math test, the Idaho Standard Achievement Test, is adminis- 
tered to all students in grades 3-8 and grade 10 and is not vertically aligned. The state 
sets cutscores for each grade level. For this study, which examined correlations between 
teacher survey responses and student achievement variables within a school, the percent- 
age of students who scored proficient or above was used to represent student achievement 
in each school. According to the latest state report card (2009/10), nearly all (99.2 percent) 
of Idaho’s students participated in state reading and math testing and received a score of 
“proficient” or “not proficient.” 

The Idaho State Department of Education tracks both average daily student attendance 
and total student enrollment at each school. To calculate an attendance rate, the school’s 
average daily enrollment was divided by the total enrollment. The Idaho State Department 
of Education also tracks the number of students tested in each school and the number of 
tested students eligible for free or reduced-price lunch. The percentage of students eligible 
for free or reduced-price lunch in each school was calculated by dividing the number of 
tested students eligible for free and reduced-price lunch by the total number of students 
tested. This was done separately for the reading and math tests. 

School-level proficiency rates are an important measure of performance, because they are 
major indicators in the state’s accountability system. In addition, the state offers capacity- 
building services specifically to schools that do not meet state targets for proficiency rates. 
Providers of capacity-building services then work with schools in the hope of increasing 
proficiency rates (Lane, 2010). 

School-level attendance rates are also an important measure of student outcomes in this 
study. Several studies have found that attendance mediates student performance (Conard, 
2006; Duran-Narucki, 2008). In other words, better attendance contributed to better 
student performance, while poor attendance contributed to poor performance. While 
attendance is not an outcome that providers of capacity-building services seek to affect 
directly, high attendance in Idaho schools may be related to better student performance. 
Therefore, this study investigated how EES data in Idaho related to attendance rates. 

School-level rates of eligibility for free or reduced-price lunch represented an important 
covariate in this study, serving as a proxy for the level of need of the student body. Indeed, 
several studies have confirmed the link between school-level percentages of students eli- 
gible for free or reduced-price lunch and student outcomes (Kahlenberg, 2001). Because 
of this well researched link, this variable was included in the data examination. In addi- 
tion, when Bylsma (2008) examined a variety of school-level variables, he found that free 
or reduced-price lunch eligibility showed several statistically significant correlations with 
the nine dimensions. In elementary and middle schools eight of nine school-level means 
for the dimensions were significantly correlated with percentages of students eligible for 
free or reduced-price lunch, while in high schools one of the nine school-level means for 
the dimensions was significantly correlated with percentages of students eligible for free or 
reduced-price lunch. 

Participating schools 

The study sample included all schools in the 2012 EES database that were designated by 
CEE as receiving capacity-building services. This database contained 75 such schools, 
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which represented 97 percent of the 77 schools that could possibly have received capacity- 
building services for schools in need of improvement in 2012. Possible participation was 
determined by examining several Idaho State Department of Education lists of participat- 
ing schools, such as those available on the department’s website or from department staff. 
These lists did not always have the same total number of schools, which made determining 
the actual number of participating schools difficult. However, two schools that typically 
appeared on department lists were missing from the EES data. These two schools notwith- 
standing, the EES data include more schools than the department lists, but the additional 
schools varied somewhat by list. 

It is assumed that the Idaho State Department of Education lists dropped schools that 
various department officials believe may not have fully participated in capacity-building 
services for schools in need of improvement. However, the criteria for inclusion on the 
lists varied and in some cases were not stated. By contrast, schools in the CEE data had 
their survey administration and scoring paid for by the Idaho State Department of Educa- 
tion, and CEE believed these schools had received at least some capacity-building services. 
Therefore, for the sake of clear definition, all 75 schools in the 2012 EES data are included 
in the analyses. A subset of 33 of these schools also had EES data for 2010 and 2011. 

All teachers in each school were invited to participate in the EES. A total of 1,745 teachers 
from 75 schools responded in 2012; 830 teachers from 33 schools responded in 2011; and 
820 teachers from the same 33 schools responded in 2010 (table B2). The total teacher 
response rate was 91 percent in 2012 for all schools. No school with a below-average return 
rate had fewer than four teachers who responded. 

The data also included school-level information on reading and math proficiency, atten- 
dance, and percentage of students eligible for free or reduced-price lunch. Almost all 
schools (more than 90 percent) had data for all variables (table B3). When proficiency rates 
and free or reduced-price lunch rates were missing, they were missing because the school 
was too small for the data to be publicly reported or, in one instance, because the data 
were missing from the publicly available dataset. 


Table B2. School and teacher response rates on the Educational Effectiveness 


Survey were similar in 2010, 2011, and 2012 


Full dataset (77 

33 schools 

33 schools 

33 schools 


schools) in 2012 

in 2010 

in 2011 

in 2012 

Variable 

Number 

Percent 

Number Percent 

Number 

Percent 

Number 

Percent 

School response rate 

75 

97 

33 100 

33 

100 

33 

100 

Total teacher responses 

1,745 

91 

820 87 

830 

86 

863 

89 

Range of teacher 
responses across schools 

4-73 

58-100 

4-64 31-100 

5-72 

60-100 

5-73 

63-100 

Standard deviation of 
teacher response rate 
across schools 

14 

11 

13 14 

13 

12 

15 

11 


Note: Data on the number of teachers per school are from the Common Core of Data because neither the 
Center for Educational Effectiveness nor the Idaho State Department of Education tracks that information. 

Source: Data from the Educational Effectiveness Survey (2010, 2011, and 2012); U.S. Department of Educa- 
tion, 2011, 2012. 
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Table B3. Almost all participating schools had publicly available data, 2010, 2011, 


and 2012 

Variable 

75 schools 
in 2012 

33 schools 
in 2010 

33 schools 
in 2011 

33 schools 
in 2012 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Number 

Percent 

Reading proficiency 

73 

97 

32 

97 

32 

97 

32 

97 

Math proficiency 

73 

97 

32 

97 

32 

97 

32 

97 

Eligibility for free or 









reduced-price lunch 

74 

99 

32 

97 

32 

97 

32 

97 

Attendance 

70 

93 

31 

94 

— 

— 

29 

88 


— is not available because data were lost in a data storage system conversion. 
Source: Idaho State Department of Education, n.d. b, c, d. 


All attendance data were missing for 2011 because the Idaho State Department of Educa- 
tion lost the data in a data storage system conversion. 

In 2010 and 2012 more schools were missing data on attendance than were missing data 
on the other variables. This was due in part to the way the attendance rate was calculat- 
ed: the average daily attendance divided by the official enrollment, which is established 
in October of each school year. Several of the schools with missing data were alterna- 
tive schools with rolling admission, which resulted in attendance rates of greater than 
100 percent. These schools were dropped from the analyses. 

Data collection 

Regional Educational Laboratory Northwest, CEE, and the Idaho State Department of 
Education complied with all applicable federal and state laws and regulations protecting 
the privacy of study participants, including the requirements of the Family Educational 
Rights and Privacy Act. All parties agreed that all personally identifiable information 
exchanged would be protected, stored, disposed of, and otherwise kept confidential. 

Federal regulations require data suppression to protect privacy. For example, “states are 
required to define a minimum number of students in a reporting group or subgroup 
required to publish results consistent with the protection of personally identifiable infor- 
mation” (Seastrom, 2010). The presentation of data in this report was reviewed to ensure 
that privacy was protected, and the two organizations that provided data reviewed the 
report for compliance with the minimum number of participants required for reporting. 
Further, this report does not present any data at the individual or school level. Instead, 
study results (means, standard deviations, correlations) were reported for the entire dataset 
of 75 schools, for subsets of the secondary and elementary schools within this dataset, and 
for the subset of 33 schools with multiple years of data. A full copy of the Regional Educa- 
tional Laboratory Northwest data security guidelines is available on request. 

Regional Educational Laboratory Northwest created a data-sharing agreement with CEE 
and received the EES data in a secure data transfer on May 14, 2012. The schooLlevel state 
test proficiency data (Idaho State Department of Education, n.d. b), the data for calculat- 
ing percentages of students eligible for free or reduced-price lunch (Idaho State Depart- 
ment of Education, n.d. d), and total school enrollment figures (Idaho State Department 


B-6 




of Education, n.d. c) were all obtained from the Idaho State Department of Education 
website. Average daily attendance data (Idaho State Department of Education, n.d. a) on 
the department website were aggregated to the district level. SchooUevel average daily 
attendance data were obtained from the Idaho State Department of Education by request 
on December 21, 2012. All datasets included school identification variables, which were 
used to merge the data. 

Study methods 

The data were analyzed using multiple steps. First, descriptive statistics were used to 
explore the EES data and the publicly available data. Next, to describe the EES data at the 
teacher level and the creation of teacherdevel EES dimensions, frequencies, means, ranges, 
standard deviations, and Cronbach’s alphas were used. 

Means, standard deviations, and histograms were used to describe the EES data after the 
dimensions were aggregated to the school level. This was done for the following groups of 
schools: 

• The 75 schools in the total sample in 2012. 

• The 39 secondary and 35 elementary schools in 2012. 3 

• The 33 schools with data for 2010, 2011, and 2012. 

Means, standard deviations, and histograms for percentages of students proficient in 
reading and math, percentages of students eligible for free or reduced-price lunch, and 
attendance rates were also calculated for these schools. 

To address the research question about the connection between EES data and publicly 
available student outcome data, scatterplots on which the x axis was the schoohlevel mean 
of each EES dimension and the y axis was the publicly available variables were exam- 
ined, and corrections for nonlinear relationships and outliers were considered. The scatter 
graphs showed either no relationship between the variables or positive linear relationships. 
Then, a series of Pearson’s correlations and linear regressions was conducted. 

The scatterplots did identify some potential outliers in the variables representing reading 
proficiency, math proficiency, and attendance. Box plots were used to further examine 
the distributions for these variables. Potential outliers were cases with values lower than 
1.5 times the interquartile range. (No cases were above 1.5 times the interquartile range.) 
In the overall dataset of 75 schools in 2012, 4 schools had attendance rates lower than 
this quantity. In both reading and math five schools had proficiency rates lower than 
this quantity, but only three of these schools had the same rate in reading and math. No 
schools identified by this means as outliers in attendance were also outliers in reading or 
math. 

It was first determined that these outliers did not result from errors in data entry. Whether 
the outliers were particular types of schools that might reasonably be expected to have 
lower proficiency and attendance rates, such as alternative schools or special education 
schools, was checked. Attendance outliers contained no detectable patterns. In reading 
and math proficiency rates some outliers were alternative schools. These schools might be 
expected to have lower proficiency rates. 
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It was ultimately decided to leave all the schools in the analyses for two reasons. First, no 
pattern or reason that the schools were lower in achievement or attendance was detected. 
Second, the educators tasked with improving schools were interested in serving low-achiev- 
ing schools, such as the ones identified as outliers by the statistical rule of thumb described 
above. 

Pearson’s correlations were calculated for the three sets of schools mentioned above and 
the three outcomes: reading proficiency rates, math proficiency rates, and attendance rates. 
To guard against error associated with multiple comparisons, the Benjamini'Hochberg 
correction was used for each set of schools separately by outcome (Benjamini & Hochberg, 
1995). For example, for the 75 schools in the total sample in 2012, the Benjamini'Hochberg 
correction was used for the correlations between the dimensions and reading proficiency; 
a separate Benjamini'Hochberg correction was then used for the correlations between the 
dimensions and math proficiency, and finally a third Benjamini'Hochberg correction was 
used for attendance rates. 

In the series of regressions the outcomes of interest were percentage meeting reading profF 
ciency, percentage meeting math proficiency, and mean attendance rate. As with the cor- 
relations, the Benjamini'Hochberg correction was used to guard against error associated 
with multiple comparisons. The outcomes are represented by y i in the equation below. 
Each model used a different predictor of interest centered on the grand mean, including 
the schooHevel mean EES score for each of the nine dimensions, as well as the mean for 
the total items. The regression model was: 

y t = b 0 +bj[Mean EES dimension score] i + b 2 [% free or reduced-price lunch] + e. 

For ordinary least squares regression to yield a consistent estimate of the relationship 
between EES and student outcome (b j), EES should not correlate with the error term (e ; ). 
Students’ need for teacher attention, direction, and support was a latent characteristic of 
the student body and was likely to affect both the teacher perception of the school (EES) 
and the student outcome (y). For this reason a variable was included that served as a proxy 
for this latent characteristic. 

Among available variables, the percentage of students eligible for free or reduced-price 
lunch appeared to be the natural choice for the proxy. Students from low-income house- 
holds could be expected to need more teacher attention, direction, and support than other 
students do, making their teachers’ work more challenging. Past research has found that 
the percentage of students eligible for free or reduced-price lunch correlates with student 
outcomes (Kahlenberg, 2001). In addition, this variable correlated consistently with the 
EES dimensions in Bylsma’s study (2008). 
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Appendix C. Descriptive statistics 


This appendix discusses the descriptive statistics for the full sample of 75 schools with 
2012 data and for the 33 schools for which Educational Effectiveness Survey (EES) data 
were available for 2010 and 2011 as well as 2012. It also discusses the descriptive analyses 
for the publicly available data. 

Educational Effectiveness Survey 

Nearly all teachers rated nearly all the items on a five-point Likert scale. For one item 
under the dimension “clear and shared focus,” one respondent indicated “no opinion.” This 
response was dropped from the analyses. 

Teacher responses to items skewed toward the positive side of the scale (table Cl). Aver- 
aged across all items: 

• 34 percent were “5 — almost always true.” 

• 37 percent were “4 — often true.” 

• 21 percent were “3 — sometimes true.” 

• 6 percent were “2 — seldom true.” 

• 3 percent were “1 — almost never true.” 


Table Cl. Percentage distribution across response categories for items on the Educational 
Effectiveness Survey were clustered at the high end of the scale, 2012 


Dimension 

Item 

Almost 

always 

true 

Often true 

Sometimes 

true 

Seldom 

true 

Almost 
never true 

Clear and shared 
focus 

Important decisions here are based on the 
mission/purpose of this school. 3 

36 

44 

17 

2 

1 


1 understand the mission/purpose of our school. 

57 

33 

7 

2 

1 


Staff 1 work with demonstrate commitment to our 
mission. 

37 

45 

16 

2 

0 


My work contributes to the mission/purpose of 
this school. 

66 

31 

3 

0 

0 


My performance goals are set based on the 
mission/purpose of this school. 

40 

42 

14 

3 

1 


The mission/vision of this school and district are 
aligned with each other. 

40 

41 

15 

2 

1 


This school has a data-driven school improvement 
plan. 

52 

36 

10 

1 

1 


(continued) 
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Table Cl. Percentage distribution across response categories for items on the Educational 
Effectiveness Survey were clustered at the high end of the scale, 2012 (continued) 

Dimension 

Item 

Almost 

always 

true 

Often true 

Sometimes 

true 

Seldom 

true 

Almost 
never true 

High standards and 
expectations for all 

Staff 1 work with demonstrate commitment to 
quality education. 

49 

40 

11 

1 

0 

students 

1 believe that all students can meet state reading 
standards. 

24 

38 

22 

8 

8 


Our staff believes that all students can meet 
state reading standards. 

19 

43 

25 

7 

6 


1 believe that all students can meet state math 
standards. 

19 

42 

24 

8 

7 


We hold one another accountable for student 
learning. 

26 

41 

25 

7 

1 


Our staff believes that all students can meet 
state math standards. 

16 

42 

29 

7 

6 


We hold one another accountable for behavior 
that is culturally sensitive. 

19 

35 

32 

11 

3 

Effective school 
leadership 

1 actively participate in the evaluation of my 
performance objectives. 

46 

38 

12 

3 

1 


My principal is committed to quality education. 

66 

25 

7 

2 

1 


Staff members at all levels are treated fairly here. 

34 

35 

20 

7 

4 


1 am comfortable presenting new ideas to my 
principal. 

45 

30 

17 

5 

4 


My principal cares about me as a person. 

55 

26 

12 

4 

3 


My principal has a student-learning focus. 

54 

33 

9 

3 

1 


My principal is comfortable presenting new ideas 
to the staff. 

56 

30 

10 

3 

1 


My principal facilitates systems/processes to 
support school improvement. 

41 

40 

13 

3 

1 


My principal listens to my ideas and concerns. 

53 

28 

12 

4 

3 


1 talk with my principal/administrator about 
progress on performance goals. 

28 

36 

23 

8 

4 

Curriculum, 
instruction, and 

This school provides a curriculum that is relevant 
and meaningful. 

37 

45 

15 

3 

1 

assessment 
aligned with 

Instruction is personalized to meet the needs of 
each student. 

19 

46 

30 

4 

1 

standards 

Common assessments are used to inform 
instruction. 

33 

41 

22 

3 

1 


Regular assessment is used to monitor student 
progress. 

49 

41 

9 

1 

0 


Our staff demonstrates a thorough understanding 
of state learning standards for reading. 

31 

42 

22 

4 

2 


The reading program we teach is aligned with 
state learning standards. 

46 

36 

13 

3 

2 


All teachers integrate literacy and numeracy 
concepts into their teaching. 

27 

40 

26 

6 

2 


Our staff demonstrates a thorough understanding 
of state learning standards for math. 

28 

41 

24 

6 

1 


The math program we teach is aligned with the 
state learning standards. 

47 

36 

13 

3 

1 


This district uses assessments aligned to 
standards and instruction. 

42 

42 

13 

2 

1 

(continued) 
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Table Cl. Percentage distribution across response categories for items on the Educational 
Effectiveness Survey were clustered at the high end of the scale, 2012 (continued) 


Dimension 

Item 

Almost 

always 

true 

Often true 

Sometimes 

true 

Seldom 

true 

Almost 
never true 

Frequent 
monitoring of 

Data from peer observations leads to meaningful 
change in instructional practice. 

18 

31 

28 

13 

10 

learning and 
teaching 

We are frequently informed about how well we are 
doing. 

24 

36 

27 

9 

4 


Assessment data is used to identify student 
needs and appropriate instructional intervention. 

47 

39 

11 

2 

1 


Struggling students receive early intervention and 
remediation to acquire skills. 

34 

39 

22 

4 

1 


We monitor the effectiveness of instructional 
interventions. 

34 

44 

18 

4 

1 


We reflect upon instructional practice to inform 
our conversations about improvement. 

28 

45 

22 

4 

1 


We are encouraged to participate in classroom 
observation. 

35 

29 

21 

8 

6 


Teachers collaboratively plan lessons. 

18 

35 

31 

11 

5 

Focused 

professional 

1 participate in a professional learning community 
focused on improving student learning. 

47 

37 

14 

2 

1 

development 

The training 1 have been to in this district helps 
me do my job better. 

26 

38 

26 

7 

2 


My principal (or administrator) talks to me about 
my professional development. 

30 

34 

24 

9 

4 


Appropriate data are used to guide building- 
directed professional development. 

26 

41 

23 

8 

2 


We are provided training to collaborate on 
improving student learning. 

29 

38 

24 

6 

2 


We have opportunities to learn effective teaching 
strategies for the cultures represented in our 
school. 

14 

31 

35 

14 

5 


Our teachers engage in classroom-based 
professional development activities (e.g. peer 
coaching) that focus on improving instruction. 

23 

33 

29 

10 

5 


Our teachers engage in professional development 
activities to learn and apply reading skills and 
strategies. 

27 

36 

26 

8 

3 


Our teachers engage in professional development 
activities to learn and apply math skills and 
strategies. 

28 

38 

25 

7 

2 


We are provided training to support a culturally 
responsive learning environment. 

15 

29 

34 

16 

7 


(continued) 
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Table Cl. Percentage distribution across response categories for items on the Educational 
Effectiveness Survey were clustered at the high end of the scale, 2012 (continued) 


Dimension 

Item 

Almost 

always 

true 

Often true 

Sometimes 

true 

Seldom 

true 

Almost 
never true 

High levels of 
collaboration and 

There is a willingness to address conflict in this 
school. 

26 

37 

26 

8 

3 

communication 

When there is a problem in my school, we talk 
about how to solve it. 

23 

38 

28 

7 

4 


When staff members attend conferences/ 
seminars, they share what they learned. 

19 

37 

30 

9 

5 


Staff at this school collaborate to improve 
student learning. 

43 

38 

15 

3 

1 


Collaboration between the district and schools is 
based upon trust and respect. 

18 

39 

30 

8 

4 


Students understand the expectations and 
standards of this school. 

21 

49 

25 

5 

1 


Staff in our school are consistently truthful. 

33 

44 

18 

3 

1 


Parents and community understand the 
expectations and standards of this school. 

14 

38 

36 

10 

3 


There is effective 2-way communication between 
the district and our school. 

18 

38 

29 

9 

5 


Our school meets regularly to monitor 
implementation of our school improvement plan. 

42 

36 

15 

5 

2 


We collaboratively plan the integration of literacy 
and numeracy concepts across the curriculum. 

20 

37 

28 

11 

4 


Staff in our school do not manipulate others to 
achieve their goals. 

38 

38 

17 

4 

3 

Supportive learning 

1 have a good or best friend at work. 

53 

26 

13 

5 

4 

environment 

There is someone at work with whom 1 confide. 

55 

24 

13 

4 

4 


Confidential information is carefully guarded in 
this school. 

40 

39 

16 

3 

2 

Our staff can count on one another for help when 
needed. 

48 

37 

13 

2 

1 

This school is orderly and supports learning. 

38 

46 

14 

2 

1 

There are people here who care about me as a 
person. 

50 

33 

14 

2 

1 

1 am encouraged to learn and grow in my school. 

47 

35 

13 

3 

1 

We honor agreements made with each other. 

43 

39 

14 

2 

2 

Students in this school are engaged in learning. 

24 

50 

23 

2 

1 

Staff members enforce consistent behavior 
expectations and consequences in their classrooms. 

23 

43 

26 

6 

2 

1 receive recognition or praise for a job well done. 

26 

31 

26 

10 

7 

We have a system for celebrating student success. 

38 

34 

21 

5 

2 

Our staff will “go the extra mile" for others. 

45 

38 

14 

2 

1 

1 understand and apply concepts of cultural 
responsiveness in my daily work. 

28 

42 

24 

5 

1 

This school reviews and addresses issues of 
cultural responsiveness. 

11 

31 

36 

16 

6 

Staff in this school can depend on one another. 

47 

35 

14 

2 

1 


(continued) 
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Table Cl. Percentage distribution across response categories for items on the Educational 
Effectiveness Survey were clustered at the high end of the scale, 2012 (continued) 


Dimension 

Item 

Almost 

always 

true 

Often true 

Sometimes 

true 

Seldom 

true 

Almost 
never true 

High levels 

This school encourages parent involvement. 

37 

41 

18 

3 

1 

of family and 
community 

Our teachers effectively communicate student 
progress to parents. 

29 

49 

20 

2 

0 

involvement 

This school has activities to celebrate the 
cultures of its community. 

12 

26 

35 

21 

6 


For important decisions, we collaborate with 
parents and the community. 

18 

36 

31 

11 

4 


This school communicates effectively with 
families of all cultures. 

22 

38 

32 

6 

2 


The curriculum we teach reflects the cultures of 
the community we serve. 

16 

34 

36 

11 

3 

Average across all items 

34 

37 

21 

6 

3 


a. One respondent (.058 percent) indicated “no opinion — not applicable,” but the other percentages were not affected. 
Source: Authors' analysis of data from the Educational Effectiveness Survey (2012). 
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A teacherdevel dimension score was calculated for each of the nine dimensions and for all 
the items together. This dimension score was the mean of the response values (1-5) for the 
items corresponding to each dimension. The internal consistency of each dimension was 
then described using Cronbach’s alpha (table C2). These alphas ranged from .84 to .98, 
signaling levels of internal consistency justifying use of an instrument (Hair et al., 2006). 


Table C2. All mean scale scores on the Educational Effectiveness Survey in schools receiving 
capacity-building services rounded to 4 — often true, 2012 


Dimension 

Mean 

Minimum 

Maximum 

Standard 

deviation 

Cronbach’s 

alpha 

Number 
of items 

Number of 
teachers 

Clear and shared focus 

4.28 

1.00 

5.00 

.5689 

.84 

7 

1,745 

Effective school leadership 

4.18 

1.00 

5.00 

.7167 

.91 

10 

1,744 

Curriculum, instruction, and 
assessment aligned with standards 

4.05 

1.00 

5.00 

.5994 

.88 

10 

1,744 

Supportive learning environment 

4.04 

1.00 

5.00 

.5847 

.89 

16 

1,745 

Frequent monitoring of learning and 
teaching 

3.83 

1.43 

5.00 

.6809 

.84 

8 

1,744 

High levels of collaboration and 
communication 

3.78 

1.55 

5.00 

.6509 

.88 

12 

1,745 

High standards and expectations for 
all students 

3.73 

1.43 

5.00 

.7373 

.85 

7 

1,743 

Focused professional development 

3.72 

1.20 

5.00 

.7293 

.89 

10 

1,744 

High levels of family and community 
involvement 

3.68 

1.00 

5.00 

.7083 

.85 

6 

1,743 

Total items 

3.94 

2.00 

5.00 

.5589 

.98 

86 

1,745 

Note: Scale scores range from 1 to 5. Data cover 1,745 teachers in 75 schools. 
Source: Authors’ analysis of data from the Educational Effectiveness Survey (2012). 
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Finally, descriptive statistics aggregated across the 75 schools in 2012 were calculated in 
two steps. The items were averaged across teachers, and then the teacher average respons- 
es were aggregated across each school (table C3). Overall, these descriptive analyses 
showed that the survey dimensions had a restricted range (not all possible values of the 
variables were present). Further, responses were skewed to the positive end of the scale. 
When rounded to the nearest whole number, most responses clustered around “4 — often 
true,” and no school rated any dimension less than “3 — sometimes true.” This restriction 
was due in part to the aggregation to the school level. However, the schooHevel analysis 
was important to Idaho education leaders and to others interested in teacher perception 
surveys, because they typically work to improve the school overall, not just some groups of 
students. 


Table C3. All mean Educational Effectiveness Survey dimension scores for all schools and for 
grade-level groupings rounded to “4 — often true,” 2012 



All schools (n = 

75) 

Secondary schools (n 

= 39) 

Elementary schools (n 

= 35) 

Dimension 

Mean 

Standard 

deviation 

Range 

Mean 

Standard 

deviation 

Range 

Mean 

Standard 

deviation 

Range 

Clear and shared focus 

4.31 

.24 

3.79- 

4.81 

4.24 

.23 

3.79- 

4.81 

4.39 

.24 

3.92- 

4.79 

Effective school leadership 

4.19 

.36 

3.22- 

4.78 

4.14 

.39 

3.22- 

4.78 

4.25 

.31 

3.47- 

4.78 

Curriculum, instruction, and 
assessment aligned with 
standards 

4.10 

.29 

3.44- 

4.76 

4.01 

.27 

3.44- 

4.76 

4.21 

.28 

3.72- 

4.62 

Supportive learning 
environment 

4.07 

.28 

3.33- 

4.76 

4.02 

.29 

3.33- 

4.76 

4.12 

.27 

3.50- 

4.53 

Frequent monitoring of 
learning and teaching 

3.83 

.36 

2.75- 

4.51 

3.72 

.38 

2.75- 

4.41 

3.96 

.29 

3.33- 

4.51 

High levels of collaboration 
and communication 

3.81 

.31 

3.22- 

4.60 

3.77 

.33 

3.22- 

4.60 

3.87 

.28 

3.38- 

4.40 

High standards and 
expectations for all students 

3.79 

.33 

3.16- 

4.63 

3.72 

.30 

3.16- 

4.63 

3.88 

.33 

3.21- 

4.53 

Focused professional 
development 

3.74 

.35 

2.71- 

4.45 

3.65 

.36 

2.71- 

4.45 

3.85 

.29 

3.24- 

4.35 

High levels of family and 
community involvement 

3.70 

.35 

3.00- 

4.62 

3.64 

.33 

3.00- 

4.62 

3.76 

.36 

3.09- 

4.39 

Total items 

3.96 

.28 

3.30- 

4.62 

3.90 

.29 

3.30- 

4.62 

4.04 

.26 

3.56- 

4.52 


Note: The number of secondary and elementary schools does not sum to 75 because one school served grade spans that included 
both elementary and secondary and was thus dropped from this analysis. Rows are ordered from highest to lowest means for all 
schools. 

Source: Authors' analysis of data from the Educational Effectiveness Survey (2012). 
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In addition, descriptive statistics were calculated for the 33 schools for which EES data 
were available for 2010 and 2011 as well as 2012 (table C4). As with the EES dimensions 
overall, these descriptive statistics showed some restriction of range that was unavoidable 
due to aggregation to the school level. Means were lower for earlier years of survey partici- 
pation and lowest in 2010. 


Table C4. All mean Educational Effectiveness Survey dimension scores for schools with three years of 
data rounded to “4 — often true,” 2010, 2011, and 2012 




2010 



2011 



2012 


Dimension 

Mean 

Standard 

deviation 

Range 

Mean 

Standard 

deviation 

Range 

Mean 

Standard 

deviation 

Range 

Clear and shared focus 

4.19 

0.23 

3.73- 

4.62 

4.31 

0.22 

3.87- 

4.81 

4.35 

0.25 

3.79- 

4.79 

Effective school leadership 

4.11 

0.31 

3.52- 

4.76 

4.17 

0.33 

3.29- 

4.83 

4.18 

0.40 

3.27- 

4.78 

Curriculum, instruction, and 
assessment aligned with 
standards 

4.03 

0.26 

3.64- 

4.69 

4.11 

0.30 

3.58- 

4.86 

4.11 

0.29 

3.70- 

4.76 

Supportive learning 
environment 

3.97 

0.26 

3.34- 

4.58 

4.02 

0.28 

3.46- 

4.88 

4.05 

0.29 

3.40- 

4.53 

Frequent monitoring of 
learning and teaching 

3.67 

0.36 

2.75- 

4.35 

3.78 

0.37 

2.93- 

4.68 

3.88 

0.38 

2.75- 

4.51 

High levels of collaboration 
and communication 

3.68 

0.29 

2.87- 

4.24 

3.78 

0.29 

3.23- 

4.39 

3.83 

0.32 

3.22- 

4.40 

High standards and 
expectations for all students 

3.62 

0.22 

3.12- 

4.03 

3.76 

0.32 

3.20- 

4.73 

3.82 

0.35 

3.16- 

4.63 

Focused professional 
development 

3.61 

0.32 

3.03- 

4.22 

3.73 

0.30 

3.16- 

4.28 

3.81 

0.34 

3.03- 

4.35 

High levels of family and 
community involvement 

3.62 

0.30 

3.10- 

4.40 

3.65 

0.36 

2.96- 

4.88 

3.72 

0.37 

3.00- 

4.39 

Total items 

3.85 

0.24 

3.32- 

4.36 

3.94 

0.26 

3.43- 

4.69 

3.98 

0.31 

3.30- 

4.52 


Source: Authors’ analysis of data from the Educational Effectiveness Survey (2010, 2011, and 2012). 
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Publicly available data 


The descriptive analysis of the publicly available data from 2012 showed that mean pro- 
ficiency rates ranged from 81 percent to 91 percent (table C5). Attendance rates ranged 
from 92 percent to 94 percent, and the percentage of students eligible for free or reduced- 
price lunch ranged from 61 percent to 62 percent. The Idaho State Department of Educa- 
tion provided data on the percentage of students eligible for free and reduced-price lunch 
separately for reading and math because a slightly different set of students participated 
in these state tests at some schools. This was likely due to student absences and to some 
special education students taking alternative tests for one subject but not the other. 


Table C5. Mean school-level student outcomes varied by grade level, but the percentage of students 
eligible for free or reduced-price lunch rates did not, 2012 




All schools (n = 75) 


Secondary schools (n 

= 39) 

Elementary schools (n 

= 35) 

Variable 

Number 
of schools 
with data 

Mean 

(percent) 

Standard 

deviation 

(percent) 

Range 

(percent) 

Number 
of schools 
with data 

Mean 

(percent) 

Standard 

deviation 

(percent) 

Range 

(percent) 

Number 
of schools 
with data 

Mean 

(percent) 

Standard 

deviation 

(percent) 

Range 

(percent) 

Proficient in reading 

73 

91 

5.6 

62-98 

37 

91 

5.0 

75-96 

35 

90 

6.4 

62-98 

Proficient in math 

73 

85 

8.8 

52-98 

37 

81 

9.6 

52-95 

35 

89 

5.6 

71-98 

Attendance 

70 

93 

4.8 

70-99 

35 

92 

4.8 

76-99 

34 

94 

4.8 

70-98 

Eligible for free or reduced- 
price lunch (reading test) 

74 

61 

16.1 

0-89 

38 

61 

13.5 

0-82 

35 

62 

18.9 

0-89 

Eligible for free or reduced- 
price lunch (math test) 

74 

61 

16.0 

0-89 

38 

61 

13.3 

0-78 

35 

62 

18.8 

0-89 


Note: The number of secondary and elementary schools does not sum to 75 because one school served grade spans that included 
both elementary and secondary and was thus dropped from this analysis. Rows are ordered from highest to lowest means for all 
schools. 

Source: Authors’ analysis of data from Idaho State Department of Education (n.d. a, b, c, d). 
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The publicly available variables for the subset of 33 schools with three years of EES data 
showed means and standard deviations similar to those for the sample of all 75 schools in 
2012 (table C6). Means for proficiency rates were lower in 2010 than in 2012. Attendance 
remained stable, while free and reduced-price lunch rates fluctuated. 

Histograms were created for the variables shown in tables C5 and C6. Like the histograms 
of the EES dimensions, these descriptive statistics showed that not all possible values of 
the variables were present. Again, this is due in part to aggregation to the school level. 
However, aggregation to the school level was essential in answering the question of this 
study. 


Table C6. Mean school-level reading and math proficiency and eligibility for free or reduced-price lunch 
increased between 2010 and 2012, while attendance remained stable 




2010 



2011 



2012 


Variable 

Number 
of schools 
with data 

Mean 

(percent) 

Standard 

deviation 

(percent) 

Range 

(percent) 

Number 
of schools 
with data 

Mean 

(percent) 

Standard 

deviation 

(percent) 

Range 

(percent) 

Number 
of schools 
with data 

Mean 

(percent) 

Standard 

deviation 

(percent) 

Range 

(percent) 

Proficient in reading 

32 

89 

7.5 

60-98 

32 

91 

4.2 

79-97 

32 

92 

4.5 

75-97 

Proficient in math 

32 

84 

12.6 

34-96 

32 

86 

8.8 

49-96 

32 

86 

9.2 

52-96 

Attendance 

31 

93 

5.0 

70-99 

— 

— 

— 

— 

29 

93 

6.3 

70-99 

Eligible for free or reduced- 
price lunch (reading test) 

32 

60 

19.6 

0-89 

32 

66 

10.1 

47-87 

32 

65 

10.5 

45-85 

Eligible for free or reduced- 
price lunch (math test) 

32 

60 

19.5 

0-89 

32 

66 

10.1 

47-87 

32 

65 

10.2 

45-85 


— is not available because data were lost in a data storage system conversion. 

Source: Authors' analysis of data from Idaho State Department of Education (n.d. a, b, c, d). 
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Notes 


1. In conducting this analysis, the Benjamini-Hochberg correction was employed to 
guard against error associated with multiple comparisons (Benjamini & Hochberg, 
1995), meaning that the relationships between variables could have occurred by 
chance. Because this study was not experimental, it cannot demonstrate that any vari- 
able caused achievement or attendance to increase. 

2. Other uses of the EES perception data — such as measuring teacher satisfaction with 
the school environment — might be appropriate if these uses do not assume that the 
survey results relate directly to student achievement. However, these other uses were 
not examined in this study. 

3. Elementary and secondary schools do not sum to 75 because one school’s grade-level 
configuration did not fit into either category and was thus not included. 
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